Search AI Products and News

On December 5th, the Byte Bean Bag model team launched the latest code model evaluation benchmark - FullStack Bench, covering over 11 real-world scenarios, supporting 16 programming languages, and including 3374 questions. Compared to previous evaluation standards, this benchmark can more accurately assess the code development capabilities of large models across a broader programming domain, facilitating the optimization of models in real-world programming tasks. Current mainstream code evaluation benchmarks, such as HumanEval and MBPP, typically focus on basic and advanced.

Anthropic announced on Monday the launch of a new initiative aimed at funding the development of new benchmarks capable of assessing the performance and impact of artificial intelligence models, including its own generative model, Claude. According to information released in Anthropic's official blog, the company will provide financial support to third-party organizations to develop tools that "effectively measure the advanced capabilities of artificial intelligence models." Interested organizat

ZhiYuan Research Institute has released a code generation training dataset called TACO, aimed at providing more challenging training data and evaluation benchmarks for code generation models. TACO has advantages in terms of data scale, quality, and evaluation schemes, including a larger training and testing set, diverse problem-solving answers, and fine-grained labels. Experimental results show that current popular code generation models show significant differences compared to GPT-4 in TACO evaluations, indicating that there is still room for improvement in this field. TACO is not just a challenging...

["Zhipu AI has released an evaluation benchmark AlignBench for Chinese large models", "AlignBench can evaluate the alignment level of models and human intentions in multiple dimensions", "The dataset is divided into 8 major categories, including knowledge Q&A, writing generation, role-playing, and various types of questions", "Developers can use AlignBench for evaluation and score using a robust scoring model", "By logging into the AlignBench website, users can submit their evaluations."]

SuperCLUE has released the August rankings for Chinese large models, featuring 5 different ranking evaluations that selected 16 general large language models, utilizing 3,337 new test questions. The performance gap between domestic large models on Chinese tasks and GPT-3.5 is narrowing.